COMMENT ⊗   VALID 00005 PAGES
C REC  PAGE   DESCRIPTION
C00001 00001
C00002 00002	.NEWSEC (EXTENSIONS TO HAL,VISUAL FEEDBACK)
C00013 00003	.newss DYNAMIC FRAMES 
C00019 00004	.NEWSS (EXTENSIONS TO OTHER ARMS AND DEVICES,EXTENSIONS)
C00021 00005	.NEWSS FINE CONTROL,FINE CONTROL
C00023 ENDMK
C⊗;
.NEWSEC (EXTENSIONS TO HAL,VISUAL FEEDBACK)
.NEWSS (INCORPORATING VISUAL FEEDBACK,VISUAL FEEDBACK)
.NEWSSS NECESSARY CAPABILITIES

	This is a list of capabilities which would have to be implemented
in order to do dynamic visual feedback within HAL.

  PICTURE BUFFERS AND WINDOWS

	We need a new datatype, PICTURE, to contain a digitized picture,
information on the camera used (particularly its location and orientation),
what lens was used, what filters, and perhaps other information.

Subpictures, that is windows, should be extractible from the picture itself,
so that a visual processing routine can look at whatever part it needs.

 CAMERA CONTROL

	There should be a syntax for specifying how to move a camera
    to a desired location.  For example,
.UNFILL
	AIM CAMERA_1 AT (30, 40, 10)
		USING LENS=2, FILTER=CLEAR, IRIS=2.8;
.REFILL

	There should be a syntax for specifying that a picture be taken
    and stored in a certain picture buffer.  Since cameras have their own
    built-in synchronization, detailed timing control may be complicated.
    Read the EXPLICIT CONTROL OF TIMING section below for some more ideas
    on this subject.

 OBJECT MODELS

	There should be sufficiently powerful data structures (such as
    arrays and list structures) available to implement complex object
    descriptions such as a network of features.

	It should be possible to implement programs which find
predicted objects in pictures by use of modeling information.
This may involve the use of recursion and backup, neither of
which is currently available.

  VISUAL PROCESSING PRIMITIVES

	There should be a mechanism for calling PDP11 and SPS41
    routines which share data such as pictures and object models.
(The SPS41 is a signal processor which we will use for some vision
work.)
To an extent, this already exists with the EXTERNAL PDP11 procedure.

MOTIONS OF ACCOMODATION

	There should be a way of specifying how to servo an arm
    based upon visual, force, or tactile information.
The arm is expected to change its trajectory as a function of
sensory input; this would allow visual servoing, for example.
An implementation would
    involve dynamically changing the arm's destination or dynamically
    specifying relative changes to be made.  In either case, time is
    an important variable.  Consider a typical sequence of events:
.BEGIN PREFACE 0; NARROW 8,8;TABBREAK
	    (1) A picture is taken of the arm.
	    (2) The picture is analyzed to determine an arm correction.
	    (3) While the visual processing is being done, the arm
		continues to move.  Hence a prediction should be made
		and incorporated into the specified correction.
	    (4) The correction is sent to the servo.
.END

 EXPLICIT CONTROL OF TIMING

	As pointed out above, time is an important factor within visual
    feedback.  To do visual feedback the user almost has to
    deal with the scheduling problems at a different level than
    COBEGIN-COEND or even EVENTs.  There should be a way of accessing
    the clock and a way of guaranteeing a certain amount of time
    for visual processing once a picture has been taken.

	It would also be useful to separate the `setup'
    for a MOVE from the actual beginning of a move.  This suggests
    a setup and trigger mechanism to squeeze as much processing
    as possible into "free" PDP11 time.

 INTERACTIVE DESIGN OF VISUAL PROCESSING

	There should be an interface to a graphics system such as
    Bruce Baumgart's GEOMED [Baumgart] so the user can symbolically position
    the assembly parts and cameras, simulate arm motions, and 
    extract potential object models from synthetic pictures.
    The system supervisor should be flexible enough to allow the
    user to interactively manipulate the actual arms and cameras
    so that the resulting TV pictures correspond with the synthetic
    views.  This involves consistent models of the world.


.NEWSSS STAGES IN INCORPORATING VISUAL FEEDBACK

	There are roughly three different stages in the process
of incorporating visual feedback into HAL: (1) completely separate
modules, (2) picture taking within HAL but models and processing separate,
and (3) everything in HAL.  These stages are briefly discussed below.

.GROUP
    COMPLETELY SEPARATE MODULES

	This means that the object modules, interpreters, camera
    control routines, etc. are in one or more modules and the HAL system
    is another module.  Communication between modules is by
    messages.  This type of communication restricts the mode
of operation; feedback will only be available while the arm is not
in motion.  Motions of accomodation would not be possible.
.APART

	The current Stanford hand-eye system is of this form.  It
    will be straightforward to provide this type of system with HAL.
    However, it has obvious limitations and hopefully would only
    be a temporary solution.

    PICTURE TAKING WITHIN HAL

	This is the first step toward a complete integration.
HAL would provide camera control, pictures, picture taking, and
    ways to call procedures which share data.  The object models could
    either be written in SAIL (and be on the PDP10) of be written in
    a PDP11 language (and be on the PDP11).  In either case the models
    and pictures would be available to external routines which analyze
    pictures and return improved location values for objects.  Visual
    servoing and dynamic feedback still could not be done;  there is no
    way to control the scheduling to insure the necessary service.

	This type of procedure-calling is designed into the current
    HAL system.  It mainly involves a smart loader.  The other extensions
    are reasonably straightforward; it appears to be an easy step
    up to this type of system.  Its advantage over the previous system
    is that the basic requirements for doing visual feedback are all
    directly accessible from within one system (assuming the routines
    are on the PDP11).  This provides a chance to try out some of the
    ideas before moving on to the next stage.

    COMPLETE INTEGRATION

	Complete integration would involve motions of accomodation
in full generality, with modifications to trajectories while they
are being executed.  Picture taking and processing would all be
run under HAL, and they would be interfaced into the timing scheme
to insure proper service.  Not only would true visual servoing
be possible, but also fine control of the hand based on delicate
touch sensing.

.newss DYNAMIC FRAMES 

One very desirable feature would be an ability to describe
and use continually varying variables.
In industrial applications, for instance,
it would be very nice if the 
runtime system could automatically make the corrections required
to track an object on a moving conveyor.
Initially, this facility is not being implemented, although
we are studying the problems involved.  Actually, only a very
few new constructs would need to be introduced into the language
to allow such things to be described.  The principal addition
required is a way of warning the compiler that some variables
may be "dynamic".  For instance,
.unfill
	DYNAMIC VECTOR v;
	DYNAMIC FRAME chain_hoist;
.refill
would tell the compiler that v and conveyor_belt may vary continuously
with time.  Trajectories to any locations that depend on such
variables must be handled a bit differently by the servo.  Instead
of applying the brakes or continuing to null out a set of joint
angles and forces at the end of a MOVE, the servo should 
periodically recompute the destination location, obtain a new
arm solution, and then servo to the new joint values.

The normal HAL graph structures are readily adapted to such dynamic
values.  Essentially all that is required is the addition of a
special reserved scalar variable TIME, which is changed every
clock tick, thus invalidating any values calculated by expressions
that depend on TIME (see {ssref gph} and {ssref gpa}).  For
instance we might have
.unfill
	DYNAMIC FRAME conveyor_belt;
	SCALAR speed; α{speed of the conveyor beltα}
	UNITS SECONDS,INCHES;
	speed α←5;  α{i.e., 5 inches/secondα}

	conveyor_belt <= [(0,speed*TIME,0):(0,0,0)]
.maybreak

.COMT 12
      α{ In this example, we won't ever use the "true" location
	of the belt.  Rather, we will ATTACH things to it,
	so that they are carried along by the belt. α}
.END

.maybreak
	SCALAR t0;
	REQUIRE "PUMP.075" SOURCE_FILE;
.COMT 12
      α{ This defines, among other things, the frames
	pumpbase and pumpgrasp.  Initially, suppose that
	we know that the pumpbase is somewhere on the 
	conveyor.  We call a vision routine to find it. α}
.END

	VISUALLY_LOCATE(pumpbase,t0);
	
.COMT 12
      α{ Also, set t0 to the value of TIME at which the picture 
	was taken (ie the time that the pumpbase was at the 
	location set by the procedure) α}
.END

.maybreak
	ATTACH pumpbase TO conveyor_belt
		AT (pumpbase α→[(0,speed*t0,0):(0,0,0)]);

      α{ One effect of this is:
	   pumpbase(t) <= (pumpbase(t0)α→conveyor(t0))*conveyor;
	α}
.maybreak

	MOVE YELLOW TO pumpgrasp;

.COMT 12
      α{ Presumably, pumpgrasp is attached rigidly to pumpbase.
	Since pumpbase is attached to a dynamic thing (conveyor_belt)
	then pumpgrasp is computed dynamically, too, so that
	the arm will track the grasp point α}
.END
.maybreak

	CENTER YELLOW;
      α{ grasps the object α}
.maybreak
	DETACH pumpbase FROM conveyor_belt;
	ATTACH pumpbase TO YELLOW;
	MOVE pumpbase TO jig_location_1; α{ wherever %4that%* isα}
.refill

It is perhaps worth pointing out that there is nothing particularly
magical about TIME; a similar technique could be used, say, for
moving to some frame whose value is a function of a continuously
varying A-to-D reading.
.NEWSS (EXTENSIONS TO OTHER ARMS AND DEVICES,EXTENSIONS)

	The initial version of the HAL system will be designed to run
with two  Stanford Arms,  but the system is  in no way limited to any
particular manipulators.    All manipulator-dependent  routines  are
grouped together  and are  written in  SAIL;   in order  to interface
another  manipulator to  the system these  routines would  have to be
rewritten, most notably  the solution and  dynamics models.  In  the
case of  non-manipulator type devices,  such  as cranes,
the trajectory generating routines would also need rewriting, but as
we lack any experience in this direction we will pursue it no further
at this time.

	Simple devices such as vices or tools have their own keyword
syntax and are controlled by the OPERATE statement. In this  case new
routines would need to be added.

.NEWSS FINE CONTROL,FINE CONTROL

	Interactive control of the  arm has to date been  limited; we
can  output joint  torque and  monitor joint  position, and  have two
binary touch sensors inside  the fingers. Force-sensing elements  are
being developed for  the hand and we are interested  in more powerful
touch  sensors; when we have gained  experience with these devices we
will extend the language to facilitate their use. The present version
of  the language  reflects  those things  which we  have  verified in
practice and feel will move development in the right direction.

.NEWSS COLLISION AVOIDING,COLLISION AVOIDING

	Since the  available collision avoiders  are quite  slow, the
initial  system relies  upon the  user to  provide her  own collision
avoiding in  the form  of VIAs  and DEPROACHes.  When fast  collision
avoiders become  available they can  be meaningfully included  in the
system.